If the downtown area of a major U.S. city is growing, there is little chance that it’s also providing affordable housing for all income groups.